{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Celebrity Recognition using Amazon Rekognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "This notebook provides a walkthrough of [celebrity recognition API](https://docs.aws.amazon.com/rekognition/latest/dg/celebrities.html) in Amazon Rekognition. You can quickly identify well known people in your video and image libraries to catalog footage and photos for marketing, advertising, and media industry use cases.\n",
    "\n",
    "* https://docs.aws.amazon.com/rekognition/latest/dg/celebrities.html\n",
    "* https://docs.aws.amazon.com/rekognition/latest/dg/API_RecognizeCelebrities.html\n",
    "* https://docs.aws.amazon.com/rekognition/latest/dg/API_StartCelebrityRecognition.html\n",
    "* https://docs.aws.amazon.com/rekognition/latest/dg/API_GetCelebrityRecognition.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "from IPython.display import HTML, display, Image as IImage\n",
    "from PIL import Image, ImageDraw, ImageFont\n",
    "import time\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "role = sagemaker.get_execution_role()\n",
    "bucket = sagemaker_session.default_bucket()\n",
    "region = boto3.Session().region_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rekognition = boto3.client(\"rekognition\")\n",
    "s3 = boto3.client(\"s3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p ./tmp\n",
    "temp_folder = \"tmp/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recognize Celebrities in Images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "imageName = \"content-moderation/media/GrandTourjc.png\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(IImage(url=s3.generate_presigned_url(\"get_object\", Params={\"Bucket\": bucket, \"Key\": imageName})))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Call Amazon Rekognition to recognize celebrities in the image\n",
    "https://docs.aws.amazon.com/rekognition/latest/dg/API_RecognizeCelebrities.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "recognizeCelebritiesResponse = rekognition.recognize_celebrities(\n",
    "    Image={\n",
    "        \"S3Object\": {\n",
    "            \"Bucket\": bucket,\n",
    "            \"Name\": imageName,\n",
    "        }\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Review JSON response returned by Rekognition Celebrity Recognition API\n",
    "In the JSON response below, you will see CelebrityFaces which contains information about recognized celebrities.\n",
    "\n",
    "For each recognized celebrity, you will see information like Name, Id, Urls and additional information about their facial attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(recognizeCelebritiesResponse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Show Image with Bounding Boxes Around Recognized Celebrities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def drawBoundingBoxes(sourceImage, boxes):\n",
    "    # blue, green, red, grey\n",
    "    colors = ((255, 255, 255), (255, 255, 255), (76, 182, 252), (52, 194, 123))\n",
    "\n",
    "    # Download image locally\n",
    "    imageLocation = temp_folder + os.path.basename(sourceImage)\n",
    "    s3.download_file(bucket, sourceImage, imageLocation)\n",
    "\n",
    "    # Draws BB on Image\n",
    "    bbImage = Image.open(imageLocation)\n",
    "    draw = ImageDraw.Draw(bbImage)\n",
    "    width, height = bbImage.size\n",
    "    col = 0\n",
    "    maxcol = len(colors)\n",
    "    line = 3\n",
    "    for box in boxes:\n",
    "        x1 = int(box[1][\"Left\"] * width)\n",
    "        y1 = int(box[1][\"Top\"] * height)\n",
    "        x2 = int(box[1][\"Left\"] * width + box[1][\"Width\"] * width)\n",
    "        y2 = int(box[1][\"Top\"] * height + box[1][\"Height\"] * height)\n",
    "\n",
    "        draw.text((x1, y1), box[0], colors[col])\n",
    "        for l in range(line):\n",
    "            draw.rectangle((x1 - l, y1 - l, x2 + l, y2 + l), outline=colors[col])\n",
    "        col = (col + 1) % maxcol\n",
    "\n",
    "    imageFormat = \"PNG\"\n",
    "    ext = sourceImage.lower()\n",
    "    if ext.endswith(\"jpg\") or ext.endswith(\"jpeg\"):\n",
    "        imageFormat = \"JPEG\"\n",
    "\n",
    "    bbImage.save(imageLocation, format=imageFormat)\n",
    "\n",
    "    display(bbImage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "boxes = []\n",
    "celebrities = recognizeCelebritiesResponse[\"CelebrityFaces\"]\n",
    "for celebrity in celebrities:\n",
    "    boxes.append((celebrity[\"Name\"], celebrity[\"Face\"][\"BoundingBox\"]))\n",
    "\n",
    "drawBoundingBoxes(imageName, boxes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recognize Celebrities in Videos\n",
    " Celebrity recognition in video is an async operation. \n",
    " https://docs.aws.amazon.com/rekognition/latest/dg/API_StartCelebrityRecognition.html\n",
    " - We first start celebrity recognition job which returns a Job Id.\n",
    " - We can then call `get_celebrity_recognition` to get the job status and after job is complete, we can get celebrity metadata.\n",
    " - In production use cases, you would usually use StepFucntion or SNS topic to get notified when job is complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "videoName = \"content-moderation/media/GrandTour720.mp4\"\n",
    "strDetail = \"Celebrites detected in video<br>=======================================<br>\"\n",
    "strOverall = \"Celebrities in the overall video:<br>=======================================<br>\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3FilePrefix = \"https://s3.amazonaws.com\"\n",
    "if not region == \"us-east-1\":\n",
    "    s3FilePrefix = \"https://s3-{}.amazonaws.com\".format(region)\n",
    "\n",
    "s3VideoUrl = \"{0}/{1}/{2}\".format(s3FilePrefix, bucket, videoName)\n",
    "\n",
    "videoTag = \"<video controls='controls' autoplay width='640' height='360' name='Video' src='{0}'></video>\".format(\n",
    "    s3VideoUrl\n",
    ")\n",
    "\n",
    "videoui = \"<table><tr><td style='vertical-align: top'>{}</td></tr><tr><td>{}</td></tr></table>\".format(\n",
    "    videoTag, strDetail\n",
    ")\n",
    "\n",
    "display(HTML(videoui))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Call Rekognition to Start a Job for Celebrity Rekognition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "startCelebrityRekognition = rekognition.start_celebrity_recognition(\n",
    "    Video={\n",
    "        \"S3Object\": {\n",
    "            \"Bucket\": bucket,\n",
    "            \"Name\": videoName,\n",
    "        }\n",
    "    },\n",
    ")\n",
    "\n",
    "celebrityJobId = startCelebrityRekognition[\"JobId\"]\n",
    "display(\"Job Id: {0}\".format(celebrityJobId))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wait for Celebrity Rekognition Job to Complete\n",
    "* In production use cases, you would usually use StepFucntion or SNS topic to get notified when job is complete.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "getCelebrityRecognition = rekognition.get_celebrity_recognition(JobId=celebrityJobId, SortBy=\"TIMESTAMP\")\n",
    "\n",
    "while getCelebrityRecognition[\"JobStatus\"] == \"IN_PROGRESS\":\n",
    "    time.sleep(5)\n",
    "    print(\".\", end=\"\")\n",
    "\n",
    "    getCelebrityRecognition = rekognition.get_celebrity_recognition(JobId=celebrityJobId, SortBy=\"TIMESTAMP\")\n",
    "\n",
    "display(getCelebrityRecognition[\"JobStatus\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Review JSON response returned by Rekognition Celebrity Recognition API\n",
    "In the JSON response below, you will see list Celebrities which contains information about recognized celebrities.\n",
    "\n",
    "For each recognized celebrity, you will see information like Timestamp, Name, Id, Urls\n",
    " and additional information about their facial attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(getCelebrityRecognition)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Display Names of Recognized Celebrities in the Video\n",
    "Display timestamps and celebrites detected at that time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "theCelebs = {}\n",
    "\n",
    "# Celebrities detected in each frame\n",
    "for celebrity in getCelebrityRecognition[\"Celebrities\"]:\n",
    "    if \"Celebrity\" in celebrity:\n",
    "        cconfidence = celebrity[\"Celebrity\"][\"Confidence\"]\n",
    "        if cconfidence > 95:\n",
    "            ts = celebrity[\"Timestamp\"]\n",
    "            cname = celebrity[\"Celebrity\"][\"Name\"]\n",
    "            strDetail = strDetail + \"At {} ms: {} (Confidence: {})<br>\".format(ts, cname, round(cconfidence, 2))\n",
    "            if not cname in theCelebs:\n",
    "                theCelebs[cname] = cname\n",
    "\n",
    "\n",
    "# Unique faces detected in video\n",
    "for theCeleb in theCelebs:\n",
    "    strOverall = strOverall + \"Name: {}<br>\".format(theCeleb)\n",
    "\n",
    "# Display results\n",
    "display(HTML(strOverall))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Detect Non-Celebrities\n",
    "Now let us try an image with non-celebrities in the image.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "customCelebrityImageName = \"content-moderation/media/chris-antje.png\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(\n",
    "    IImage(url=s3.generate_presigned_url(\"get_object\", Params={\"Bucket\": bucket, \"Key\": customCelebrityImageName}))\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Call Amazon Rekognition to recognize celebrities in the image\n",
    "\n",
    "customCelebrityResponse = rekognition.recognize_celebrities(\n",
    "    Image={\n",
    "        \"S3Object\": {\n",
    "            \"Bucket\": bucket,\n",
    "            \"Name\": customCelebrityImageName,\n",
    "        }\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Display Rekognition Response\n",
    "You will see Rekognition return an empty list for CelebrityFaces and \n",
    "\n",
    "UnrecognizedFaces list with unrecognized faces that were detected in the image.\n",
    "\n",
    "In the next module you will learn how to get custom-celebrity faces recognized.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(customCelebrityResponse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Show Image and Bounded Boxes around Detected Faces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cboxes = []\n",
    "faces = customCelebrityResponse[\"UnrecognizedFaces\"]\n",
    "for face in faces:\n",
    "    cboxes.append((\"Unrecognized Face\", face[\"BoundingBox\"]))\n",
    "\n",
    "drawBoundingBoxes(customCelebrityImageName, cboxes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Congratulations!\n",
    "You have successfully used Amazon Rekognition to identify celebrities in images an videos. \n",
    "\n",
    "# References\n",
    "- https://docs.aws.amazon.com/rekognition/latest/dg/celebrities.html\n",
    "- https://docs.aws.amazon.com/rekognition/latest/dg/API_RecognizeCelebrities.html\n",
    "- https://docs.aws.amazon.com/rekognition/latest/dg/API_StartCelebrityRecognition.html\n",
    "- https://docs.aws.amazon.com/rekognition/latest/dg/API_GetCelebrityRecognition.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Release Resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "\n",
    "<p><b>Shutting down your kernel for this notebook to release resources.</b></p>\n",
    "<button class=\"sm-command-button\" data-commandlinker-command=\"kernelmenu:shutdown\" style=\"display:none;\">Shutdown Kernel</button>\n",
    "        \n",
    "<script>\n",
    "try {\n",
    "    els = document.getElementsByClassName(\"sm-command-button\");\n",
    "    els[0].click();\n",
    "}\n",
    "catch(err) {\n",
    "    // NoOp\n",
    "}    \n",
    "</script>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%javascript\n",
    "\n",
    "try {\n",
    "    Jupyter.notebook.save_checkpoint();\n",
    "    Jupyter.notebook.session.delete();\n",
    "}\n",
    "catch(err) {\n",
    "    // NoOp\n",
    "}"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
